What dynamic neural codes <br> for efficient visual processing - Colloque : "CODAGES ET REPRESENTATIONS", MASTER DE NEUROSCIENCES 2ème année - <a href="http://invibe.net/LaurentPerrinet/Presentations/2017-11-15_ColloqueMaster">http://invibe.net/LaurentPerrinet/Presentations/2017-11-15_ColloqueMaster</a>

What dynamic neural codes
for efficient visual processing ?

Laurent U Perrinet, INT

Acknowledgements:

Jim Bednar @ Continuum Analytics & University of Edinburgh
Rick Adams and Karl Friston @ UCL - Wellcome Trust Centre for Neuroimaging
Wahiba Taouali, Giacomo Benvenuti, Frédéric Chavane - ANR BalaV1
Mina Aliakbari Khoei, Guillaume Masson and Anna Montagnini - FACETS-ITN Marie Curie Training

Colloque : "CODAGES ET REPRESENTATIONS", MASTER DE NEUROSCIENCES 2ème année - http://invibe.net/LaurentPerrinet/Presentations/2017-11-15_ColloqueMaster - November 15th, 2017

(AUTHOR) Hi, I am Laurent Perrinet from (LOGO) the Institute de Neurosciences de la Timone in Marseille, a joint unit from the CNRS and AMU.
(OBJECTIVE) in this seminar, I will be focus in highlighting some key challenges in understandiung neural codes for efficient visual processing and
(ACKNO) this endeavour involves different techniques and tools ... From the head on, I wish to thanks people who collaborated and in particular .. mostly funded by the ANR bala V1 + LONDON (Jim Bednar, Friston) + many thanks to the people setting up this seminar
(SHOW TITLE) I am interested in the link between the neural code and the structure of the world. in particular, for vision, I am researching the relation between the functional (in terms of the inferential processes leading to behaviour) organization (anatomy and activity) of low-level visual areas (V1) and the structures of natural scenes, that is of the images that hit the retina and which are relevant to visual perception in general.

This seminar is an exercise to introduce the AMU masters into the format of international conferences. As such, we will try to introduce new concepts and results which will not be found in textbooks. for this introductory seminar, I wish to to give you an overlook on some techniques used to understand how we may progress in our understanding of the neural code that I will applky to vision:

can we derive learning rules to get the most of visual information? in particular, is it possible to do it with a priori labelling of objects (aka unsupervised learning)
using such coding can we use it on real images and perform usual tasks like classification (supervised learning). this topic is today highly explored and used in all deep learning techinques
we will extend such models to incorporate constraints from real neural systems, that is, that information is dynamic and may come with delays.

Outline

Learning sparse codes: competition vs cooperation

Sparse codes for real-life images

Categorizing using sparse edges

Dynamics: Sensing with delays

A model compensating for delays

Results: the Flash-lag effect

LP (2010) Neural Computation, http://invibe.net/LaurentPerrinet/Publications/Perrinet10shl / github

This result is obtained by resolved an efficient codding problem. For your interest, I will show the formulas which allow to obtain such a result:

please do not focus on the details as it the results from years of research from the community, but just try to get the main idea
first, we can use probabilistic formalism to retrieve hidden structure in the flow of pixels : crucial here is the use of Bayes formula. This mathematical is very compact but very useful to exactly state the problem:

it basically says that we can compute a belief about some hidden parameter (for instance the intensity, orientation, scale of edges in an image) (the a postertiori probability ) knowing some data I (the input image) and also some parameters Phi which define these edges (think of them as the current structure of the brain, the synapses)

This belief is equal to the likelihood of this image times the prior

the likelihood is a measure which rates the probability of our observation assuming that we know the hypothesis a. this is computationnally

the prior is some knowledge on the data that was formed before acquiring this instance of the data (it is a priori, hence the name)

LP (2015) "Sparse models" in Biologically Inspired Computer Vision / github

Outline

Learning sparse codes: competition vs cooperation

Sparse codes for real-life images

Categorizing using sparse edges

Dynamics: Sensing with delays

A model compensating for delays

Results: the Flash-lag effect

LP (2015) "Sparse models" in Biologically Inspired Computer Vision / github

LP (2015) "Sparse models" in Biologically Inspired Computer Vision

So we found a sparse code for natural images ... but what can we do with that? Could we learn more elaborate schemes in early visual areas?

... let's get inspired by the statistics of natural images:

(natural) For instance, oriented edges that constitute images of natural scenes tend to be aligned in co-linear or co-circular arrangements, such as when 💠 you follow the contours of these boulders: lines and smooth curves are more common than other possible arrangements of edges. See for example the work of Mariano Sigman on co-circularity in natural images (see Sigman, 2001).
(neural) The visual system appears to take advantage of this prior information, and human contour detection and grouping performance is well predicted by what is coined an "association field" (Field et al., 1993)...

Outline

Learning sparse codes: competition vs cooperation

Sparse codes for real-life images

Categorizing using sparse edges

Dynamics: Sensing with delays

A model compensating for delays

Results: the Flash-lag effect

Bosking et al. (1997) Journal of Neuroscience

... is the set of long-range lateral connections between neurons, which could act to facilitate detection of contours matching the association field, and/or inhibit detection of other contours. To fill this role, the lateral connections would need to be orientation specific and aligned along contours, * (colin) and indeed such an arrangement has been found in treeshrew's primary visual cortex (Bosking et al., J Neurosci 17:2112-27, 1997)

(neural) if one looks at the primary visual area in the occipital lobe of the cortex using optical imaging as here in the treeshrew by Bosking and colleagues under the supervision of DF, one could represent the distributed, topographical representation of orientation selectivity. in (A) and (B) the orientation giving the most response at each cortical position is represented by hue using the code below from orange for horizontal to blue for verticals, and typical structures are magnified in (C): stripes (on the periphery) and pinwheels. You can understand this as a packing of a 3D feature space on the 2D surface of the cortex.
(method) Tree shrew orientation preference maps were obtained using optical imaging. Additionally, 540 nm light was used to map surface blood vessels used for alignment. Biocytin was then injected into a specific site in V1 and the animal was sacrificed 16 hours later. Slices of V1 were imaged to locate the biocytin bouton and the surface blood vessels. The blood vessel information was then used to align the orientation preference maps with the bouton images giving overlaid information on the underlying connectivity from the injection site on the animal. The original experiment used a total of ten cases.
(lateral) we show here one result of Bosking which overlay over a map or orientation selectivity the network of lateral connectivity originating froma group of neurons with similar orientations and position. There is a structure in this connectivity towards locality (more pronounced for site B) + connecting iso orientations even on long ranges (A). This type of structure tends to wire together those neurons that have similar orientations, indicating a prior to colinearities.

*(colin) ... Overall, a typical assumption that the role of lateral interactions is to enhance the activity of neurons which are collinear : it is the so-called association field formalized in Field 93, as was for instance modeled neurally in the work from P. Series or in this version for computer vision

(physio) is there a match of these structures with the statistics of natural images? 2) Some authors (Kisvarday, 1997, Chavane and Monier) even say it is weak or inexistent on a the scale of the area... 1: Hunt & Goodhill have reinterpreted above data and shown that there is more diversity than that -

LP and Bednar (2015) Scientific Reports, http://www.nature.com/articles/srep11400

Probability distribution function of "chevrons"

(angles) By computing measures of the independence of the different variables, we found that the probability density function of the second-order statistics of edges factorizes with on one side distance and scale and on the other side the 2 angles. The first component proved to be quite similar across both classes and the greater difference is seen for different angle configuration. As it can be reduced to 2 dimensions, we can plot the full probability as shown here by different contrast values assigned to all possible chevrons configurations, for all possible "azimuth" values $\psi$ on the horizontal axis and difference of orientation $ heta$ on the vertical axis. Such a plot most strikingly shows the difference between these 2 classes. one issue now that we can show the 2nd order statistics is to know if it would be possible to quantify such difference...
(colin) let's first replicate the result from Geisler by showing that relative to a given edge (segment in the center), what is the Here I show for each distance and angle the most probable difference of angle, showing that collinear and parallel edges predominate. (cocir) a similar pattern is observed the cocircular plot. it reproduces the results from Geisler on natural images, but laboratory environment shows a strong bias to colinearity. If we believe bosking link, obviously, this should have a consequence on antomy. thus we test whether these AF were different across different image classes.

LP and Bednar (2015) Scientific Reports, http://www.nature.com/articles/srep11400

we computed the statistics of edge co-occurrences for different categories (natural, man-made, or containing an animal) . These differed strongly, with animal images having more curved configurations. Here we show that the statistics of edge co-occurrences alone are sufficient to perform a rough yet robust (translation, scale, and rotation invariant) scene categorization. This seems a bit crazy (in particular with the current trend for deep learning) especially, knowing that most model (like the one from Serre 07) would rather consider that it is better done in 10 processing layers.
Classification results for different sets of natural images (50\% is chance level) and different classification methods. Note that for SVM, we controlled for different feature choice: first-order statistics (SVM 1), full second-order statistics (SVM 2) or the chevron map only (SVM C). Surprisingly, the results are very similar to the results of Serre 07, and we do not lose much performance by choosing the simple chevron map. First, this has direct application in machine learning as this translation- and rotation-independent classifier performs as well as more complicated networks (that is they have more parameters) - and could be imported to existing architectures

LP and Bednar (2015) Scientific Reports, http://www.nature.com/articles/srep11400

We show that this geometry alone is sufficient for categorization, and that the pattern of errors made by humans is consistent with this procedure. * To see whether the patterns of errors made by humans are consistent with our model, we studied the second-order statistics of the 50 non-animal images that human subjects in Serre et al. (2007) most commonly falsely reported as having an animal. We call this set of images the false-alarm image dataset. (Left) This chevron map plot shows the ratio between the second-order statistics of the false-alarm images and the full non-animal natural image dataset, computed as in Figure 3 (left). Just as for the images that actually do contain animals , the images falsely reported as having animals have more co-circular and converging (red chevrons) and fewer collinear and orthogonal configurations (blue chevrons). (Right) To quantify this similarity, we computed the Kullback-Leibler distance between the histogram of each of these images from the false-alarm image dataset, and the average histogram of each class. The difference between these two distances gives a quantitative measure of how close each image is to the average histograms for each class. Consistent with the idea that humans are using edge co-occurences to do rapid image categorization, the 50 non-animal images that were worst classified are biased toward the animal histogram ($d' = 1.04$), while the 550 best classified non-animal images are closer to the non-animal histogram.

... but before going further, I would like to say one word. in fact, it may not obvious to the general public that problems in artificial intelligence and more generally in understanding the brain is different from one thinks.
this is perfectly illustrated in this cartoon from the famous XKCD author Randall Munroe: it is easy for an artificial intelligence to geolocalize itself anywhere in the world (or to browse a database of zillions of go games as for alpha go), but whenever it has to something apparently simple ("is there a bird?") computers are clueless
... Eye movements are part of this category of hard systems. They can be of different nature: Most simply, saccades are an orienting behavior aiming at catching a specific position in visual space (or vergence movement in depth). if after fixating at the red fixation dot you try to catch the green dot, then you need to do such saccades, which can be very fast (up to 600 deg/s) When the dot is now moving almong a trajectory, this visual motion can are also responsible for more complex movements such as the reflexive Ocular Following Response or the voluntary Smooth Pursuit EM which aims at stabilizing the visual image on the retina. INterestingly, in this latter case the position of the eye quickly centers on the present, physical position of the dot and we will later see why this is challenging
so, present computers are a bad analogy to understand the brain and we look here at different solutions found in nature by natural selection - so thanks to the organizers for providing a forum for such endeavour - let's start by looking at anatomy:

Outline

Learning sparse codes: competition vs cooperation

Sparse codes for real-life images

Categorizing using sparse edges

Dynamics: Sensing with delays

A model compensating for delays

Results: the Flash-lag effect

The Behavioral receptive field

LP and Masson (2012) Bio Behav Reviews, http://invibe.net/LaurentPerrinet/Publications/Masson12

Montagnini, LP and Masson (2015) in Biologically Inspired Computer Vision

... we have been busy in the last couple of years to make sense of this repertoire of behaviors, in particular in light with the architecture of the brain areas involved in producing these EMs, for instance for pursuit initiation, ...

the Left-side plot (extracted from this review paper) illustrates in primates a sketch of the hierarchy of neuronal population yielding to ocular following responses from the early visual areas (retina, thalamus, primary visual cortex), finessed by processing in motion specific information in downstream sensory areas (MT for speed or MST for global motion) to the brain stem and the oculomotor muscles which tranform this neural activity in a graded motor output. As such, moving stimuli covering the retina are processed by a cascade of neuronal populations, where information is processed through feedforward, lateral and feedback interactions.
in particular, we took advantage of a Bayesian, probabilistic framework to develop an unified theory, the "behavioral receptive field" which encompasses a large range of psychophysical experiments by modeling behavioral responses from population dynamics. Such cascade implements local, context-dependent extraction of motion information… I will speak more about that on Friday for those in the summer school
today, I would like to focus on a particular problem which will help us unravel the dynamics of decision making: oculomotor delays. Indeed, one challenge for modelling is to understand EMs using AI as a problem of optimal motor control under axonal delays. The central nervous system has to contend with axonal delays, both at the sensory and the motor levels. For instance, in the human visuo-oculomotor system, it takes approximately $τ_s=50~ms$ for the retinal image to reach the visual areas implicated in motion detection, and a further $τ_m=40~ms $ to reach the oculomotor muscles.

and at this instant, the sensed physical position is lagging behind (as represented here by $τ_s \cdot v 1~m$ ), that is, approximately at $45$ degrees of eccentricity (red dotted line),
while the position at the moment of emitting the motor command will be $.8~m$ ahead of its present physical position ($τ_m \cdot v$).
As a consequence, note that the player's gaze is directed to the ball at its present position (red line), in anticipatory fashion. Optimal control directs action (future motion of the eye) to the expected position (red dashed line) of the ball in the future --- and the racket (black dashed line) to the expected position of the ball when motor commands reach the periphery (muscles). This is obviously an interesting challenge for modelling an optimal control theory.

Khoei, Masson and LP (2017) PLoS CB http://invibe.net/LaurentPerrinet/Publications/KhoeiMassonPerrinet17

so let's go back on earth

... but first let now apply this model compensating for the aforementioned visual delays using a well described visual illusion: the flash-lag effect:

a first stimulus moves continuously across the screen along the central horizontal axis. In the FLE, as this moving stimulus reaches the center of the screen, a second stimulus is flashed just above it and in perfect vertical alignment. Despite the fact that the respective horizontal positions of each stimulus are physically identical when the flash occurs, the moving stimulus is most often perceived ahead of the flashed one.

debate for 80 years revived recently

motion extrapolation
differential latency
post-diction

we propose to extend the hypothesis previously proposed by Nihjawan that this effect is caused by the extrapolation of the stimulus' motion to compensate for the neural delay. However, this hypothesis was challenged by other hypothesis that this effect is due to either anatomy (differential latencies) or to the way visual awareness processes the sequence of events (the post-diction from Eagleman)

As a matter of fact, the motion extrapolation hypothesis was challenged because you can notice that the FLE is still present at initiation of the movement but this effect is not seen if the moving dot abruptly stops at the moment of the flash

A model compensating for delays

Khoei, Masson and LP (2017) PLoS CB http://invibe.net/LaurentPerrinet/Publications/KhoeiMassonPerrinet17

Our model is very simple: Following what Nihjawan called a "diagonal model", we have proposed to extend a motion-based predictive coding model by including the known sensory delay in the prediction-estimation loop
More formally, the estimated state vector $z = {x, y, u, v}$ is composed of the 2D position ($x$ and $y$) and velocity ($u$ and $v$) of a (possibly moving) stimulus. As such, we extend Nijhawan's diagonal model into a classical Markov chain in order to take into account the known neural delay $τ$: At time $t$, information is integrated until time $t-τ$, using a Markov chain and a model of state transitions $p(z_t |z_{t-\delta t})$ such that one can infer the state until the last accessible information $p(z_{t-τ} | I_{0:(t-τ)})$. This information can then be ``pushed'' forward in time by predicting its trajectory from $t-τ$ to $t$. Interestingly, we use the same model of state transition at the time scale of the delay, that is, $p(z_t | z_{t-τ})$.
Importantly, the model for transition is based on a simple model of the transport of visual information knowing the velocity. Indeed, knowing $z(t-\Delta t)$ we may infer the future position of this motion at time $t$ by using Newton's first law of mechanics. The precision of the state transition (as given by the shaded area) tunes the strength of this prediction as a diffusion parameter.
Such a model can be implemented in a two-layered neural network including a source (input) and a target (predictive) layer~\citep{KaplanKhoei14}. The source layer receives the delayed sensory information and encodes both position and velocity topographically on the different retinotopic maps of each layer. Notice that here, for simplicity, we show only one 2D map of the motions $(x, v)$. Crucially, the delay compensation in this motion-based prediction model, is simply implemented by connecting each source neuron to a predictive neuron corresponding to the corrected position of stimulus $(x+v\cdot τ, v)$ in the target layer. The precision of this anisotropic connectivity map can be tuned by the width of convergence from the source to the target populations. We will see on Friday that this property (that it is implemented in the connectivity) extends to more complex, multi-layered systems which may generalize to more complex trajectories.

A model compensating for delays

Khoei, Masson and LP (2017) PLoS CB http://invibe.net/LaurentPerrinet/Publications/KhoeiMassonPerrinet17

Our model is very simple: Following what Nihjawan called a "diagonal model", we have proposed to extend a motion-based predictive coding model by including the known sensory delay in the prediction-estimation loop
More formally, the estimated state vector $z = {x, y, u, v}$ is composed of the 2D position ($x$ and $y$) and velocity ($u$ and $v$) of a (possibly moving) stimulus. As such, we extend Nijhawan's diagonal model into a classical Markov chain in order to take into account the known neural delay $τ$: At time $t$, information is integrated until time $t-τ$, using a Markov chain and a model of state transitions $p(z_t |z_{t-\delta t})$ such that one can infer the state until the last accessible information $p(z_{t-τ} | I_{0:(t-τ)})$. This information can then be ``pushed'' forward in time by predicting its trajectory from $t-τ$ to $t$. Interestingly, we use the same model of state transition at the time scale of the delay, that is, $p(z_t | z_{t-τ})$.
Importantly, the model for transition is based on a simple model of the transport of visual information knowing the velocity. Indeed, knowing $z(t-\Delta t)$ we may infer the future position of this motion at time $t$ by using Newton's first law of mechanics. The precision of the state transition (as given by the shaded area) tunes the strength of this prediction as a diffusion parameter.
Such a model can be implemented in a two-layered neural network including a source (input) and a target (predictive) layer~\citep{KaplanKhoei14}. The source layer receives the delayed sensory information and encodes both position and velocity topographically on the different retinotopic maps of each layer. Notice that here, for simplicity, we show only one 2D map of the motions $(x, v)$. Crucially, the delay compensation in this motion-based prediction model, is simply implemented by connecting each source neuron to a predictive neuron corresponding to the corrected position of stimulus $(x+v\cdot τ, v)$ in the target layer. The precision of this anisotropic connectivity map can be tuned by the width of convergence from the source to the target populations. We will see on Friday that this property (that it is implemented in the connectivity) extends to more complex, multi-layered systems which may generalize to more complex trajectories.

Results using PBP

Let me show you now the results of our simulations on the standard flash-lag stimulus by first showing the results of a predictive system with no delay compensation. We here simply show the source layer in the prediction model which tries to track moving patterns but do not compensate for the known delay.
We observe that the moving dot is correctly tracked and the flash correctly detected. I have slowed down the speed of the movie to show that relative to each other, the flash and moving dot appear at the same position at the time of the flash.
The situation is different in the motion-based predictive system : in this slowed down movie, at the time the flash appears, the dot's position was predicted ahead of the position of the flash, similarly to what is reported in psychophysics.
Moreover, we have predicted a similar effect with a large range of parameters of the stimulus, such as speed or contrast. But for the moment, let us focus on the reasons why the model shows a similar behavior

Results using MBP

Let me show you now the results of our simulations on the standard flash-lag stimulus by first showing the results of a predictive system with no delay compensation. We here simply show the source layer in the prediction model which tries to track moving patterns but do not compensate for the known delay.
We observe that the moving dot is correctly tracked and the flash correctly detected. I have slowed down the speed of the movie to show that relative to each other, the flash and moving dot appear at the same position at the time of the flash.
The situation is different in the motion-based predictive system : in this slowed down movie, at the time the flash appears, the dot's position was predicted ahead of the position of the flash, similarly to what is reported in psychophysics.
Moreover, we have predicted a similar effect with a large range of parameters of the stimulus, such as speed or contrast. But for the moment, let us focus on the reasons why the model shows a similar behavior

A model compensating for delays

Khoei, Masson and LP (2017) PLoS CB http://invibe.net/LaurentPerrinet/Publications/KhoeiMassonPerrinet17

For that, we replot the movies I have just shown by showing for the dot the Histogram of the estimated positions as a function of time for the source layer (Left) and the target layer (right). The left-hand column illustrates the predictive model before delay compensation. The right-hand column illustrates the motion extrapolation model with delay compensation. Histograms of the inferred horizontal positions (blueish bottom panel) and horizontal speed (redish top panel) are shown in columns as a function of time. A darker level corresponds to a higher probability, while a light color corresponds to an unlikely estimation. In particular, we focus on three particular epochs along the trajectory, corresponding to the standard, flash initiated and terminated cycles. The timing of these epochs flashes are indicated by dashed vertical lines. In dark, the physical time and in green the delayed input knowing $τ=100~ms$.
Activity in both models shows three different phases. First, there is a rapid build-up of the precision of the target after the first appearance of the moving dot (at $t=300~ms$). Consistently with the Frölich effect, the beginning of the trajectory is seen ahead of its physical position. During the second phase, the moving dot is correctly tracked as both its velocity and position are correctly inferred. In the source layer, there is no extrapolation and the trajectory follows the delayed trajectory of the dot (green dotted line). In the target layer, motion extrapolation correctly predicts the position at the present time and the position follows the actual physical position of the dot (black dotted line). Finally, the third phase corresponds to motion termination. The moving dot disappears and the corresponding activity vanishes in the source layer at $t=900~ms$. However, between $t=800~ms$ and $t=900~ms$, the dot position was extrapolated and predicted ahead of the terminal position. At $t=900~ms$, while motion information is absent, the position information is still transiently consistent and extrapolated using a broad, centered prior distribution of speeds. Although it is less precise, this position of the dot at flash termination is therefore not perceived as leading the flash.
Interestingly, and thanks to the reviewers of the paper, we could extend our results to the estimation of the dot position from the dMBP model during the motion reversal experiment. In the motion reversal experiment, the moving dot reverses its direction at the middle of the trajectory (i.e., at $t=500~ms$, as indicated by the mid-point vertical dashed line). In the left column (target layer) and as in previous slide, we show the histogram of inferred positions during the dot motion and a trace of its position with the highest probability as a function of time. As expected, results are identical to that in the previous slide in the first half period. At the moment of the motion reversal, the model output is consistent with previous psychophysical reports. First, the estimated position follows the extrapolated trajectory until the (delayed) sensory information about the motion reversal reaches the system (at $t=600~ms$, green vertical dashed line). Then, the velocity is quickly reseted and converges to the new (reversed) motion such that the estimated position ``jumps'' to a position corresponding to the updated velocity. In the right column (smoothed layer), we show the results of the same data after a smoothing operation of $τ_s=100~ms$ in subjective time. This different read-out from the inferred positions corresponds to the behavioral results obtained in some experiments, such as that from~\citet{Whitney98}.

A model compensating for delays

Khoei, Masson and LP (2017) PLoS CB http://invibe.net/LaurentPerrinet/Publications/KhoeiMassonPerrinet17

For that, we replot the movies I have just shown by showing for the dot the Histogram of the estimated positions as a function of time for the source layer (Left) and the target layer (right). The left-hand column illustrates the predictive model before delay compensation. The right-hand column illustrates the motion extrapolation model with delay compensation. Histograms of the inferred horizontal positions (blueish bottom panel) and horizontal speed (redish top panel) are shown in columns as a function of time. A darker level corresponds to a higher probability, while a light color corresponds to an unlikely estimation. In particular, we focus on three particular epochs along the trajectory, corresponding to the standard, flash initiated and terminated cycles. The timing of these epochs flashes are indicated by dashed vertical lines. In dark, the physical time and in green the delayed input knowing $τ=100~ms$.
Activity in both models shows three different phases. First, there is a rapid build-up of the precision of the target after the first appearance of the moving dot (at $t=300~ms$). Consistently with the Frölich effect, the beginning of the trajectory is seen ahead of its physical position. During the second phase, the moving dot is correctly tracked as both its velocity and position are correctly inferred. In the source layer, there is no extrapolation and the trajectory follows the delayed trajectory of the dot (green dotted line). In the target layer, motion extrapolation correctly predicts the position at the present time and the position follows the actual physical position of the dot (black dotted line). Finally, the third phase corresponds to motion termination. The moving dot disappears and the corresponding activity vanishes in the source layer at $t=900~ms$. However, between $t=800~ms$ and $t=900~ms$, the dot position was extrapolated and predicted ahead of the terminal position. At $t=900~ms$, while motion information is absent, the position information is still transiently consistent and extrapolated using a broad, centered prior distribution of speeds. Although it is less precise, this position of the dot at flash termination is therefore not perceived as leading the flash.
Interestingly, and thanks to the reviewers of the paper, we could extend our results to the estimation of the dot position from the dMBP model during the motion reversal experiment. In the motion reversal experiment, the moving dot reverses its direction at the middle of the trajectory (i.e., at $t=500~ms$, as indicated by the mid-point vertical dashed line). In the left column (target layer) and as in previous slide, we show the histogram of inferred positions during the dot motion and a trace of its position with the highest probability as a function of time. As expected, results are identical to that in the previous slide in the first half period. At the moment of the motion reversal, the model output is consistent with previous psychophysical reports. First, the estimated position follows the extrapolated trajectory until the (delayed) sensory information about the motion reversal reaches the system (at $t=600~ms$, green vertical dashed line). Then, the velocity is quickly reseted and converges to the new (reversed) motion such that the estimated position ``jumps'' to a position corresponding to the updated velocity. In the right column (smoothed layer), we show the results of the same data after a smoothing operation of $τ_s=100~ms$ in subjective time. This different read-out from the inferred positions corresponds to the behavioral results obtained in some experiments, such as that from~\citet{Whitney98}.

A model compensating for delays

Khoei, Masson and LP (2017) PLoS CB http://invibe.net/LaurentPerrinet/Publications/KhoeiMassonPerrinet17

For that, we replot the movies I have just shown by showing for the dot the Histogram of the estimated positions as a function of time for the source layer (Left) and the target layer (right). The left-hand column illustrates the predictive model before delay compensation. The right-hand column illustrates the motion extrapolation model with delay compensation. Histograms of the inferred horizontal positions (blueish bottom panel) and horizontal speed (redish top panel) are shown in columns as a function of time. A darker level corresponds to a higher probability, while a light color corresponds to an unlikely estimation. In particular, we focus on three particular epochs along the trajectory, corresponding to the standard, flash initiated and terminated cycles. The timing of these epochs flashes are indicated by dashed vertical lines. In dark, the physical time and in green the delayed input knowing $τ=100~ms$.
Activity in both models shows three different phases. First, there is a rapid build-up of the precision of the target after the first appearance of the moving dot (at $t=300~ms$). Consistently with the Frölich effect, the beginning of the trajectory is seen ahead of its physical position. During the second phase, the moving dot is correctly tracked as both its velocity and position are correctly inferred. In the source layer, there is no extrapolation and the trajectory follows the delayed trajectory of the dot (green dotted line). In the target layer, motion extrapolation correctly predicts the position at the present time and the position follows the actual physical position of the dot (black dotted line). Finally, the third phase corresponds to motion termination. The moving dot disappears and the corresponding activity vanishes in the source layer at $t=900~ms$. However, between $t=800~ms$ and $t=900~ms$, the dot position was extrapolated and predicted ahead of the terminal position. At $t=900~ms$, while motion information is absent, the position information is still transiently consistent and extrapolated using a broad, centered prior distribution of speeds. Although it is less precise, this position of the dot at flash termination is therefore not perceived as leading the flash.
Interestingly, and thanks to the reviewers of the paper, we could extend our results to the estimation of the dot position from the dMBP model during the motion reversal experiment. In the motion reversal experiment, the moving dot reverses its direction at the middle of the trajectory (i.e., at $t=500~ms$, as indicated by the mid-point vertical dashed line). In the left column (target layer) and as in previous slide, we show the histogram of inferred positions during the dot motion and a trace of its position with the highest probability as a function of time. As expected, results are identical to that in the previous slide in the first half period. At the moment of the motion reversal, the model output is consistent with previous psychophysical reports. First, the estimated position follows the extrapolated trajectory until the (delayed) sensory information about the motion reversal reaches the system (at $t=600~ms$, green vertical dashed line). Then, the velocity is quickly reseted and converges to the new (reversed) motion such that the estimated position ``jumps'' to a position corresponding to the updated velocity. In the right column (smoothed layer), we show the results of the same data after a smoothing operation of $τ_s=100~ms$ in subjective time. This different read-out from the inferred positions corresponds to the behavioral results obtained in some experiments, such as that from~\citet{Whitney98}.

as a summary, we have shown that, evolving from other theories of prediction of post-diction, we have shown that perceptual results are well modeled by a simple theory. This theory states that the agent models (predicts) the scene at the present time (in spite of inherent but known delays) - and we called it parodiction from the greek "paros" the present time.
More generally, this novel theory proposes that generic neural computations systematically compensate neural delays to represent the predicted visual scene at the present time, though it is not yet directly experienced. a consequence of this view -that is shared among a large number of physicists- is that time as a continous line is an emerging property of the representation used by these agents and does not need to exist physically (though this rather bold statement is quite infalsifiable) - the world is latent - there is only the arrow of time (= laws of nature)
There is a fundamental limit to this theory: the agent has an uniform access to the whole field of view. With a foveal point, it may fail to track - thus we need action as an eye movement ; but this is another project :-)

What dynamic neural codes
for efficient visual processing ?

Laurent U Perrinet, INT

Acknowledgements:

Jim Bednar @ Continuum Analytics & University of Edinburgh
Rick Adams and Karl Friston @ UCL - Wellcome Trust Centre for Neuroimaging
Wahiba Taouali, Giacomo Benvenuti, Frédéric Chavane - ANR BalaV1
Mina Aliakbari Khoei, Guillaume Masson and Anna Montagnini - FACETS-ITN Marie Curie Training

Colloque : "CODAGES ET REPRESENTATIONS", MASTER DE NEUROSCIENCES 2ème année - http://invibe.net/LaurentPerrinet/Presentations/2017-11-15_ColloqueMaster - November 15th, 2017

Great; but now, what if I want to learn the right parameters instead of scanning all of them? A solution lies in the Free-energy princple from Karl Friston, and I will give here an introduction to this technique

in a first study, we have proposed that PERCEPTION (following Helmhotz 1866) is an active process of hypothesis testing by which we seek to confirm our predictive models of the (hidden) world: Active inference: (cite TITLE). In theory, one could find any prior to fit any experimental data, but the beauty of the theory comes from the simplicity of the models chosen to model the data at hand, such as saccades...
... and even better if these models may find a possible correspondance into the neural anatomy and explain some deviation to a control behaviour, such as that we modelled for understanding some aspects of the EMs of schizophrenic patients
Today, I will again focus on the problem of sensorimotor delays in the optimal control of (smooth) eye movements under uncertainty. Specifically, we consider delays in the visuo-oculomotor loop and their implications for active inference and I will present the results presented in the following paper (show TITLE).

This schematic shows the dependencies among various quantities modelling exchanges of an agent with the environment. It shows the states of the environment and the system in terms of a probabilistic dependency graph, where connections denote directed (causal) dependencies. The quantities are described within the nodes of this graph -- with exemplar forms for their dependencies on other variables.
Hidden (external) and internal states of the agent are separated by action and sensory states. Both action and internal states -- encoding a conditional probability density function over hidden states -- minimise free energy. Note that hidden states in the real world and the form of their dynamics can be different from that assumed by the generative model; (this is why hidden states are in bold. )

What dynamic neural codes for efficient visual processing ?

Laurent U Perrinet, INT

Acknowledgements:

Colloque : "CODAGES ET REPRESENTATIONS", MASTER DE NEUROSCIENCES 2ème année - http://invibe.net/LaurentPerrinet/Presentations/2017-11-15_ColloqueMaster - November 15th, 2017

Outline

Learning sparse codes: competition vs cooperation

Sparse codes for real-life images

Categorizing using sparse edges

Dynamics: Sensing with delays

A model compensating for delays

Results: the Flash-lag effect

Outline

Learning sparse codes: competition vs cooperation

Sparse codes for real-life images

Categorizing using sparse edges

Dynamics: Sensing with delays

A model compensating for delays

Results: the Flash-lag effect

Outline

Learning sparse codes: competition vs cooperation

Sparse codes for real-life images

Categorizing using sparse edges

Dynamics: Sensing with delays

A model compensating for delays

Results: the Flash-lag effect

Outline

Learning sparse codes: competition vs cooperation

Sparse codes for real-life images

Categorizing using sparse edges

Dynamics: Sensing with delays

A model compensating for delays

Results: the Flash-lag effect

The Behavioral receptive field

A model compensating for delays

A model compensating for delays

Results using PBP

Results using MBP

A model compensating for delays

A model compensating for delays

A model compensating for delays

What dynamic neural codes for efficient visual processing ?

Laurent U Perrinet, INT

Acknowledgements:

Colloque : "CODAGES ET REPRESENTATIONS", MASTER DE NEUROSCIENCES 2ème année - http://invibe.net/LaurentPerrinet/Presentations/2017-11-15_ColloqueMaster - November 15th, 2017

What dynamic neural codes
for efficient visual processing ?

What dynamic neural codes
for efficient visual processing ?